Papers with majority voting

45 papers
DialogGuard: Multi-Agent Psychosocial Safety Evaluation Interface of Sensitive LLM Responses (2026.acl-demo)

Copied to clipboard

Challenge: Existing tools do not surface subtler psychosocial harms, nor provide explainable rationales that practitioners need.
Approach: They propose an open-source system that lets practitioners inspect, stress-test, and create audit trails for prompted LLM agents across five psychosocial safety dimensions.
Outcome: The open-source DialogGuard system lets practitioners inspect, stress-test, and create audit trails for prompted LLM agents across five psychosocial safety dimensions.
Soft Self-Consistency Improves Language Models Agents (2024.acl-short)

Copied to clipboard

Challenge: Current “sample and select” methods rely on majority voting to score answers . however, when tasks have many distinct and valid answers, selection by voting requires a large number of samples.
Approach: They introduce a method that replaces SC's discontinuous scoring with a continuous score computed from model likelihoods to increase selection even when actions are sparsely distributed.
Outcome: The proposed method improves performance and efficiency on long-horizon interactive tasks by replacing SC’s discontinuous scoring with a continuous score computed from model likelihoods.
IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators (2024.findings-eacl)

Copied to clipboard

Challenge: Existing studies on social media bias detection focus on fine-tuning models specific to particular datasets and testing them on corresponding test sets.
Approach: They propose a general bias detection framework, IndiVec, built upon large language models and vector databases.
Outcome: The proposed framework outperforms baseline methods on four political bias datasets and provides explicit top-k indicators to interpret bias predictions.
A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition (2023.findings-acl)

Copied to clipboard

Challenge: Existing models for named entity recognition (NER) are based on large-scale labeled datasets, which always obtain using crowdsourcing.
Approach: They propose a CONfidence-based partial Label Learning method to integrate prior and posterior confidences for crowd-annotated named entity recognition models.
Outcome: The proposed model improves on real-world and synthetic datasets compared with baselines.
Learning Emphasis Selection for Written Text in Visual Media from Crowd-Sourced Label Distributions (P19-1)

Copied to clipboard

Challenge: Visual communication relies heavily on images and short texts to grab a viewer's attention and convey a message in the most efficient way.
Approach: They propose a model that employs end-to-end label distribution learning on crowd-sourced data and predicts a selection distribution, capturing the inter-subjectivity and ambiguity of the input.
Outcome: The proposed model captures the inter-subjectivity and ambiguity of the input and can be transformed to single-label learning by mapping labels to absolute labels via majority voting.
ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation (2023.emnlp-main)

Copied to clipboard

Challenge: Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation.
Approach: They propose to use a multi-head model to learn from disagreements in an active learning setting to identify annotators with a single head.
Outcome: The proposed model outperforms a single-head model in terms of uncertainty estimation and prediction while saving 70% of the annotation budget.
Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks (2023.eacl-main)

Copied to clipboard

Challenge: Disagreement can reflect different aspects of linguistic annotation, from annotators’ subjectivity to sloppiness or lack of context to interpret a text.
Approach: They propose a taxonomy of possible reasons leading to annotators' disagreement in subjective tasks and manually label part of a Twitter dataset for offensive language detection in english following this taxonomies.
Outcome: The proposed taxonomy of disagreements in linguistic datasets can be used to assess how accurate tweets belonging to different disagreement categories can be classified as offensive or not.
Work Hard, Play Hard: Collecting Acceptability Annotations through a 3D Game (2022.lrec-1)

Copied to clipboard

Challenge: Corpus-based studies on acceptability judgements have always been popular thanks to the release of the CoLA corpus, a large-scale corpus of sentences extracted from linguistic handbooks as examples of acceptable/non acceptable phenomena in English.
Approach: They present a 3D video game that was used to collect acceptability judgments on italian sentences and compare them with experts’ acceptability judgements.
Outcome: The proposed game compares the annotations of Italian sentences with those of experts and shows that they are more reliable than crowd-sourced annotations.
Deliberative Searcher: Improving LLM Reliability via Reinforcement Learning with Constraints (2026.acl-long)

Copied to clipboard

Challenge: Large language models with search capabilities often exhibit miscalibrated confidence, causing incorrect answers with high certainty.
Approach: They propose a reasoning-primary framework that integrates search operations into chain-of-thought generation while maintaining explicit confidence calibration.
Outcome: The proposed framework improves accuracy and reliability of large language models with search capabilities.
Domain-Specific NER via Retrieving Correlated Samples (2022.coling-1)

Copied to clipboard

Challenge: Successful Named Entity Recognition models fail on texts from some special domains, for example, Chinese addresses and e-commerce titles.
Approach: They propose to enhance NER models with correlated samples to help the text understanding . they draw correlated texts by the sparse BM25 retriever from large-scale in-domain unlabeled data .
Outcome: Empirical results show that NER models can be enhanced with correlated samples . the proposed model can be used to reason out the correct answer on hard cases .
When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models (2024.findings-naacl)

Copied to clipboard

Challenge: Recent studies suggest that self-reflective prompting can significantly enhance the reasoning capabilities of Large Language Models (LLMs).
Approach: They propose guidelines for when to implement self-reflection in Large Language Models.
Outcome: The proposed approach improves the reasoning capabilities of Large Language Models under a more stringent evaluation setting, and reduces tendency toward majority voting.
Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge (2025.acl-long)

Copied to clipboard

Challenge: Existing methods rely on majority voting or criteria expansion to capture detailed and detailed details, often leading to incomplete outcomes.
Approach: They propose a method which introduces additional crowd responses to compare with the candidate responses, thereby exposing deeper and more comprehensive details within the candidate answers.
Outcome: Experiments show that the proposed method improves evaluation reliability and achieves an average gain of 6.7% across five benchmarks.
Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding (2023.findings-acl)

Copied to clipboard

Challenge: Existing text decoding methods struggle to produce high-quality text . Greedy and beam search suffer from text degeneration and linguistic diversity issues .
Approach: They propose a family of decoding methods based on minimum bayes risk minimization to address diversity-quality trade-offs in open-ended natural-language generation.
Outcome: The proposed methods improve diversity-quality trade-offs on open-ended natural-language generation tasks.
GCoT-Decoding: Unlocking Deep Reasoning Paths for Universal Question Answering (2026.findings-acl)

Copied to clipboard

Challenge: Recent work on Chain-of-Thought reasoning requires manual prompts to guide the model.
Approach: They propose a general decoding strategy that generates CoT-style reasoning paths without prompts.
Outcome: The proposed method maintains strong performance on fixed and free QA tasks and achieves significant improvements on free qa.
The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding? (2025.findings-acl)

Copied to clipboard

Challenge: Existing approaches to self-improvement rely on external supervision signals in the form of seed data and/or assistance from third-party models.
Approach: They propose a framework for generating high-quality synthetic question-answer data in a fully autonomous manner.
Outcome: The proposed framework generates high-quality synthetic question-answer data in a fully autonomous manner.
Don’t waste a single annotation: improving single-label classifiers through soft labels (2023.findings-emnlp)

Copied to clipboard

Challenge: Existing methods for annotating data are limited by ambiguity and lack of context in data samples.
Approach: They challenge the traditional approach of annotating data by only providing a single label for each sample and annotator disagreement is discarded . instead, they use additional annotation information such as confidence, secondary label and disagreement to generate soft labels.
Outcome: The proposed method improves model performance and calibration on the hard label test set.
BanNERD: A Benchmark Dataset and Context-Driven Approach for Bangla Named Entity Recognition (2025.findings-naacl)

Copied to clipboard

Challenge: In a cross-dataset evaluation, models trained on BanNERD consistently outperformed those trained on four existing Bangla NER datasets.
Approach: They propose to use Bangla as a language to create the most extensive human-annotated and validated Bangla NLP dataset.
Outcome: The proposed method outperforms existing methods on Bangla NER datasets and performs competitively on English datasets.
Sarcasm Detection is Way Too Easy! An Empirical Comparison of Human and Machine Sarcasm Detection (2022.findings-emnlp)

Copied to clipboard

Challenge: sarcasm detection datasets focus on intended, rather than perceived sarcasm, but there is no comparison between human and machine performance.
Approach: They collect author-annotated sarcasm datasets that focus on intended, rather than perceived sarcasticism . they compare human-level benchmarks to that of state-of-the-art sarkasmatic detection systems .
Outcome: The proposed datasets compare human and machine performance on sarcastic tasks in English and Arabic.
CER: Confidence Enhanced Reasoning in LLMs (2025.acl-long)

Copied to clipboard

Challenge: Existing approaches to enhance the reliability of Large Language Models (LLMs) in complex reasoning tasks are limited by their limitations.
Approach: They propose an uncertainty-aware framework to enhance the reliability of Large Language Models . they quantify the confidence of intermediate answers and evaluate the reliability based on these confidences a way that reflects the reliability.
Outcome: The proposed approach improves accuracy of large language models in math and open-domain tasks by 7.4% and 5.8% over baseline approaches.
Forward-Backward Reasoning in Large Language Models for Mathematical Verification (2024.findings-acl)

Copied to clipboard

Challenge: Extensive experiments on six standard mathematical data sets and three LLMs show that FOBAR achieves state-of-the-art performance.
Approach: They propose to combine forward and backward reasoning to verify candidate answers . they propose to use a template to mask a number and ask the LLM to answer a backward question .
Outcome: Experiments on mathematical data show that proposed backward reasoning outperforms Self-Consistency.
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks (2024.naacl-long)

Copied to clipboard

Challenge: Existing approaches to label aggregation fail to capture subjective annotations and can lead to biases.
Approach: They propose annotator-aware representations for text for subjective classification tasks that involve learning representations of annotators.
Outcome: The proposed model improves on metrics that assess the performance on capturing individual annotators’ perspectives.
ReFT: Reasoning with Reinforced Fine-Tuning (2024.acl-long)

Copied to clipboard

Challenge: Existing approaches to improve the generalization of large language models are using Supervised Fine-Tuning (SFT) this approach does not show sufficient generalization ability because it only relies on the given CoT data.
Approach: They propose to use Chain-of-Thought annotations to train Large Language Models using supervised fine-tuning to improve generalization.
Outcome: The proposed approach outperforms SFT on GSM8K, MathQA, and SVAMP datasets and shows a superior generalization ability.
DS-MHP: Improving Chain-of-Thought through Dynamic Subgraph-Guided Multi-Hop Path (2025.findings-emnlp)

Copied to clipboard

Challenge: Existing knowledge graph methods lack adaptability in knowledge-intensive tasks with multiple entities and implicit multi-hop relations.
Approach: They propose a zero-shot framework to enhance LLM reasoning in multi-entity relation tasks.
Outcome: DS-MHP outperforms baselines and state-of-the-art methods on 12 datasets spanning commonsense, logical, symbolic, and arithmetic reasoning.
Voting or Consensus? Decision-Making in Multi-Agent Debate (2025.findings-acl)

Copied to clipboard

Challenge: Increasing the number of agents improves performance, while more discussion rounds before voting reduces it.
Approach: They propose two new methods to improve multi-agent debates by increasing agent diversity and reducing discussion rounds before voting.
Outcome: The proposed methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI.
ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation (2026.acl-long)

Copied to clipboard

Challenge: Existing approaches to selecting a single high-quality output from multiple generations are limited in their applicability and efficiency.
Approach: They propose a method that generalizes majority voting to open-ended text generation . modeX leverages structural information across multiple generation paths to select a "modal" output .
Outcome: The proposed framework outperforms standard single- and multi-path baselines in open-ended tasks.
ReasonerRank: Redefining Language Model Evaluation with Ground-Truth-Free Ranking Frameworks (2025.findings-acl)

Copied to clipboard

Challenge: Large Language Models (LLMs) are increasingly adopted across real-world applications . traditional evaluations rely on expensive, domain-specific ground-truth labels . obtaining labeled data is expensive, time-consuming, and often requires domain expertise .
Approach: They propose a ground-truth-free evaluation framework focused on reasoning consistency and instruction following.
Outcome: The proposed framework outperforms existing label-free methods, including majority voting, triplet ranking, and peer-review approaches.
Quality Control for Crowdsourced Bilingual Dictionary in Low-Resource Languages (2022.lrec-1)

Copied to clipboard

Challenge: a recent study shows that crowdsourcing is becoming mainstream to create bilingual dictionaries . the number of people who can speak multiple low-resource languages is limited and the average ability of workers is low.
Approach: They propose a method to aggregate the answers of evaluation tasks by majority voting . they use hyper questions to evaluate the reliability of workers and task-allocation method to select high-quality workers .
Outcome: The proposed method improves quality of bilingual dictionaries by integrating answers by majority voting.
Scalability of Bayesian Network Structure Elicitation with Large Language Models: a Novel Methodology and Comparative Analysis (2025.coling-main)

Copied to clipboard

Challenge: Existing methods for BN structure learning are limited by the size of the BN.
Approach: They propose a method for Bayesian Networks (BNs) structure elicitation that initializes several LLMs with different experiences and queries them to create a structure.
Outcome: The proposed method performs better than the existing method with one of the three studied LLMs, but the performance decreases with the increase in BN size.
Your Reasoning Model is Secretly a Reward Model - Optimization-Free Verification from Experience (2026.acl-long)

Copied to clipboard

Challenge: Existing verifiers operate on the surface text or on confidence proxies derived from token probabilities, which can be brittle.
Approach: They propose a training-free, non-parametric verifier that summarizes each reasoning trace by an activation delta and compares it to two class centroids computed from labeled experience.
Outcome: The proposed model improves selection and reranking on large and less-calibrated models.
CritiQ: Mining Data Quality Criteria from Human Preferences (2025.acl-long)

Copied to clipboard

Challenge: Existing methods to train language models rely on manual design, perplexity, or careful prompt engineering.
Approach: They propose a method that automatically mines criteria from human preferences for data quality with only 30 human-annotated pairs and performs efficient data selection.
Outcome: The proposed method improves on human-annotated test sets and shows high accuracy on code, math, and logic domains.
Self-Training Large Language Models with Confident Reasoning (2025.findings-emnlp)

Copied to clipboard

Challenge: Large language models generate reasoning paths before final answers, but learning such a path requires costly human supervision.
Approach: They propose a method that fine-tunes LLMs to prefer reasoning paths with high confidence . they propose 'cORE-PO' that fine tunes Lms to choose high-quality reasoning paths .
Outcome: The proposed method improves the accuracy of outputs on four in-distribution and two out-of-difference benchmarks.
USDC: A Dataset of  ̲User  ̲Stance and  ̲Dogmatism in Long  ̲Conversations (2025.findings-acl)

Copied to clipboard

Challenge: Previously, studies on stance and dogmatism in user conversations have focused on training models using annotated datasets at the post level, treating each post as independent and randomly sampling posts from conversation threads.
Approach: They build a dataset for studying user opinion fluctuations in 764 long multi-user Reddit conversation threads, called USDC.
Outcome: The proposed dataset analyzes user opinion fluctuations in 764 long multi-user Reddit conversation threads.
Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs (2026.findings-acl)

Copied to clipboard

Challenge: Existing approaches to test-time scaling are limited due to the quality of candidate responses.
Approach: They propose a new metric to quantify the relative improvement of self-refinement beyond majority voting.
Outcome: The proposed method achieves state-of-the-art performance across five benchmarks over other methods.
Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQL (2026.findings-acl)

Copied to clipboard

Challenge: Translating natural language questions into SQL is a core challenge in natural language understanding and human-computer interaction.
Approach: They propose a reinforcement learning framework and model family to generate accurate, executable SQL using a lightweight reward signal based solely on execution correctness.
Outcome: The proposed framework outperforms previous versions of 70B-class systems and achieves state-of-the-art execution accuracy across six diverse Text2SQL benchmarks.
Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory (2025.acl-long)

Copied to clipboard

Challenge: Recent studies have shown that scaling test-time compute can also effectively improve reasoning.
Approach: They propose a probabilistic method to efficiently predict scaling performance and identify the best prompting strategy under large sampling times.
Outcome: The proposed method significantly improves the scaling performance of majority voting on large language models.
Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems (2025.findings-emnlp)

Copied to clipboard

Challenge: Existing approaches to selecting reliable responses from multiple LLMs often depend on external verifiers, human evaluators, or self-consistency techniques.
Approach: They propose a calibrated log-likelihood-based selection framework to improve multi-LLM performance.
Outcome: The proposed method outperforms majority voting and exceeds self-consistency performance when using a large number of model calls.
Chronos: Learning Temporal Dynamics of Reasoning Chains for Test-Time Scaling (2026.findings-acl)

Copied to clipboard

Challenge: Existing methods for testing time scales treat reasoning traces or tokens equally, ignoring substantial variations in trajectory quality and localized logical failures.
Approach: They propose a chronological reasoning scorer that models each trajectory as a time series.
Outcome: The proposed method achieves relative improvements of 34.21% over Pass@128 and 22.70% over Maj@135 on HMMT25, highlighting its effectiveness.
TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent (2025.emnlp-main)

Copied to clipboard

Challenge: Existing work has focused on the (un)intended leakage of sensitive information through LLM outputs.
Approach: They propose a threat model that embeds context information into natural-looking outputs via linguistic steganography without requiring explicit control over inference inputs.
Outcome: The proposed model transmits 32-bit secrets with 87% accuracy on held-out prompts and can reach over 97% accuracy using majority voting across three generations.
STA-CoT: Structured Target-Centric Agentic Chain-of-Thought for Consistent Multi-Image Geological Reasoning (2025.findings-emnlp)

Copied to clipboard

Challenge: Reliable multi-image geological reasoning is essential for automating expert tasks in remote-sensing mineral exploration.
Approach: They propose a framework that orchestrates planning, execution, and verification agents to decompose, ground, and iteratively refine reasoning steps over geological and hyperspectral image sets.
Outcome: The proposed framework decomposes, ground, and iteratively refines reasoning steps over geological and hyperspectral image sets.
Can LLMs Really Judge? A Progressive Argumentation-Mining Framework for Distinguishing Understanding from Aggregation (2026.findings-acl)

Copied to clipboard

Challenge: Existing evaluations of large language models rely on dataset-based generation accuracy . however, generative correctness does not guarantee discriminative capability to verify solutions .
Approach: They propose a diagnostic framework that explicitly controls context and isolates discriminative behaviors.
Outcome: The proposed framework explicitly controls context and isolates discriminative behaviors.
Free-MAD: Consensus-Free Multi-Agent Debate (2026.findings-acl)

Copied to clipboard

Challenge: Existing multi-agent debate methods rely on multiple rounds of interaction among agents to reach consensus, and the final output is decided by majority voting in the last round.
Approach: They propose a multi-agent debate framework that eliminates the need for consensus among agents and reconstructs the debate phase by introducing anti-conformity.
Outcome: Experiments on eight benchmark datasets show that Free-MAD significantly improves reasoning performance while requiring only a single-round debate and thus reducing token costs.
TrustTable: A Neuro-Symbolic Auditing Framework for Faithful Table QA (2026.acl-long)

Copied to clipboard

Challenge: Large Language Models (LLMs)-based TableQA models exhibit unfaithful behavior where correct answers are derived through erroneous reasoning paths.
Approach: They propose a neuro-symbolic framework to audit LLM reasoning processes . it enforces factual grounding and ensures logical soundness by verifying reasoning chains .
Outcome: The proposed framework outperforms LLM judges in majority voting and rejection sampling with process supervision.
Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning (2026.acl-long)

Copied to clipboard

Challenge: RLVR is a paradigm for improving reasoning ability of large language models . but voting results often induce confirmation bias and suffer from sparse rewards .
Approach: They propose a framework integrating model confidence and dynamic subgroup partitioning to address these issues.
Outcome: The proposed framework outperforms recent baselines on multiple models and benchmarks.
Slim-SC: Thought Pruning for Efficient Scaling with Self-Consistency (2025.emnlp-main)

Copied to clipboard

Challenge: Recent studies show that Test-Time Scaling (TTS) can improve reasoning performance without retraining the model.
Approach: They propose a step-wise pruning strategy that identifies and removes redundant chains using inter-chain similarity at the thought level.
Outcome: The proposed method reduces inference latency and KVC usage by up to 45% and 26% with R1-Distill while maintaining or improving accuracy.
BanHADEX: Towards Explainable HAte Speech Detection in Bangla Using Human Annotated EXplanation (2026.acl-long)

Copied to clipboard

Challenge: Existing studies in Bangla focus on hate classification while overlooking interpretability.
Approach: They propose to create a dataset with human-annotated labels for banla that contains 19,203 YouTube comments spanning April 2024–June 2025.
Outcome: The proposed dataset outperforms existing datasets on open and closed-source LLMs on interpretability and better understanding of hate speech in linguistically rich yet under-resourced languages.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations